Vehicle detection system

ABSTRACT

A vehicle detection system 200 comprises: a receiver 202 arranged to receive data transmissions from a vehicle 210; a camera 204 arranged to capture an image of the vehicle 210; and a processing subsystem 206 arranged to implement a trained model for detecting and/or classifying vehicles in an image.The system 200 is arranged to operate in a first mode wherein: the camera 204 captures an image of a vehicle 210 and the receiver 202 receives a data transmission from the vehicle 210; and the processing subsystem 206 is arranged to use information resulting from the data transmission received and the image to train the model further. The system 200 is also arranged to operate in a second mode wherein: the camera 204 captures an image of a vehicle 210 and the receiver 202 does not receive a data transmission; and the processing subsystem 206 is arranged to use the image and the trained model to detect and/or classify the vehicle 210 in the image.

FIELD

The invention relates to a system and method for detecting vehicles.

BACKGROUND

Detecting vehicles is crucial to a number of applications. For example,it may be necessary at a tolling station to detect, classify andidentify vehicles in order to charge each passing vehicle an appropriateamount. In another application, it is desirable to detect vehicles inorder to monitor traffic on a road, for example so that assistance canbe sent to any stopped or crashed vehicles.

It may be possible to detect certain vehicles, e.g. which have beenprovided with systems capable of transmitting information (e.g.WLAN-based V2X transmissions), simply by using an appropriate receiver.However, many vehicles are not equipped with such technology. Detectingand classifying vehicles without such capabilities is challenging. Forexample, in applications in which cameras are used, human interventionis normally required to analyse images potentially containing vehiclesin order to identify the presence of a vehicle and determinecharacteristics of the vehicle where relevant. Such a set of ‘trainingimages’ which have been analysed and annotated by a human may be used totrain a system to detect and classify vehicles.

However, obtaining large numbers of training images analysed by humansis time consuming. It is further time consuming to train a system suchthat it can accurately detect and classify vehicles in any location(e.g. on a hill, a winding road) and conditions (e.g. weatherconditions, lighting conditions), as this requires an even larger numberof training images. Additionally to the time spent creating such largenumbers of training images, the training of the system itself usingthese images such that it effectively detects and classifies vehiclescan also be a long process. It is therefore very difficult to build anaccurate model which can detect and identify vehicles quickly. This canlead to either large inefficiencies in creating accurate vehicledetection systems or, on the other hand, an inadequately trained,inaccurate vehicle detection system.

An aim of the present invention is to provide an improved vehicledetection system.

SUMMARY

When viewed from a first aspect, the present invention provides avehicle detection system comprising:

-   -   a receiver arranged to receive data transmissions from a        vehicle;    -   a camera arranged to capture an image of said vehicle; and    -   a processing subsystem arranged to implement a trained model for        detecting and/or classifying vehicles in an image;

wherein the system is arranged to operate in at least

-   -   a first mode wherein:        -   the camera captures an image of a vehicle and the receiver            receives a data transmission from the vehicle; and        -   the processing subsystem is arranged to use information            resulting from the data transmission received and the image            to train the model further; and    -   a second mode wherein:        -   the camera captures an image of a vehicle and the receiver            does not receive a data transmission; and        -   the processing subsystem is arranged to use the image and            the trained model to detect and/or classify the vehicle in            the image.

When viewed from a second aspect, the present invention provides amethod of detecting vehicles, the method comprising:

-   -   capturing an image of a vehicle using a camera;    -   using a receiver to listen for a data transmission from the        vehicle;    -   determining whether said data transmission has been received        from the vehicle;    -   if said data transmission is received from the vehicle, using        said image to train a trained model further to detect and/or        classify vehicles, and    -   if no data transmission is received from the vehicle, using the        image and the trained model to detect and/or classify the        vehicle in the image.

When viewed from a third aspect, the present invention provides anon-transitory computer-readable medium having stored thereon programinstructions that, upon execution by a computing device, cause thecomputing device to:

-   -   capture an image of a vehicle using a camera;    -   listen for a data transmission from the vehicle;    -   determine whether said data transmission has been received from        the vehicle;    -   if said data transmission is received from the vehicle, use said        image to train a trained model further to detect and/or classify        vehicles, and    -   if no data transmission is received from the vehicle, use the        image and the trained model to detect and/or classify the        vehicle in the image.

Thus it will be seen by those skilled in the art that in accordance withthe present invention, information resulting from data transmitted froma vehicle is used together with captured images to provide data forimproving the trained model for detecting and/or classifying vehicles,but if no data transmission is received, the model can be used to detectand/or classify the vehicle. The may provide a robust and reliablesystem which can detect/classify vehicles based on information providedby them but which can be usefully deployed before all, or even most,vehicles are equipped to do this and also to provide a back-up where thedata transmission and/or receipt of resulting information does notfunction as intended. This may reduce the breadth of training datarequired, or eliminate the requirement for training data before avehicle detection system can be implemented.

Using image recognition to detect and/or classify vehicles may beadvantageous in requiring less physical roadside infrastructure than inprior art systems which include laser detectors which must be onseparate gantries to cameras and DSRC receivers.

The training of the model in accordance with the present invention willtypically be a machine learning process. Initial training may be carriedout using a set of images and corresponding classification informationwhich could for example be generated by human input to indicate simplywhether an image contains a relevant vehicle or not (e.g. if the modelis to be used only for detection) or to classify the vehicle (e.g.: van,lorry, motorbike, car; saloon, hatchback, 4×4, SUV; large, medium, smalletc.). Such input may also indicate whether the vehicle is facingtowards or away from the camera in the image.

In accordance with the invention further training during use of thesystem does not need to involve additional human input in the form ofanalysing and annotating large numbers of training images. Insteadfurther training can be carried out automatically while the system is inuse, as in cases where the vehicle sends a data transmission this cancontain, or be used to access, information which can be reliably used toassociate an image captured at the same time with the appropriateclassification (or simply presence) of the vehicle. This may continuallyimprove the detection and/or classification of vehicles over time, asthe number of iterations of using the transmission received from avehicle and images captured by the camera to train the model to detectand/or classify a vehicle increases.

The trained model may comprise a neural network. Basic training ofneural networks to identify vehicles from images is known in the art.For example, using neural networks to determine of license plate numberof a vehicle in an image is used widely for tolling and law enforcement.Refining the neural network by further training in accordance with theinvention may strengthen certain pathways to create a more accuratemodel.

A common problem with machine learning models is to use the right numberof parameters in the model. Too many parameters will result inoverfitting where the model works very well on the trained images, butless well in a deployed environment. Too few parameters will typicallyresult in wrong classifications. Different types of machine learningmodels will handle this problem differently.

In a set of embodiments the neural network comprises a convolutionalneural network (CNN). In a CNN one or more steps of pre-processing areapplied to reduce the data volume and the number of parameters.Convolutional neural networks may have one or more convolution layerseffectively creating one or more derived images from the input image.The convolution layer (or process) may employ a sliding window withweights that is applied to a subsection of the image to produce aweighted average value for each subsection. The process may beconsidered to be similar to how a human may focus on a part of an imageby employing a magnifying glass.

In a set of embodiments the convolution process is arranged to providean output image where the resulting size is independent of the inputimage size (i.e. camera resolution). Before the convolution process,standard machine learning procedures may include converting colour inputimages to grayscale, cropping areas where no vehicle should be detected,and normalizing grayscale values applied. Convolution filters may bearranged to detect edges and corners in the image. This means that theinner layers of the model will work on edges and corners in the originalimage and not areas of uniform colour.

In some embodiments to reduce the size of the input data and thus themodel parameters, another non-linear convolution step may be applied toextract a strongest image feature, e.g. edge or corner, in each of aplurality of subsections. While training the model, it may be importantto have a relevant cost function to measure the n-dimensional distancebetween the known classification vector and the output from the neuralnetwork. In general, neural networks will have better performance asadditional layers are added, but this requires a larger trainingdataset. In the context of the current invention, there may be a largeamount of images with known classification making it possible to benefitfrom the additional layers.

This invention may also be implemented with non-CNN model types.

The present invention may further enable improved detection of vehiclesas systems can be further trained in the relevant implementationlocation of the system (i.e. in situ). For example, a particular systemmay be located in a region that is subject to low light conditions (e.g.in a valley) or in a city where sunlight is reflected from nearbybuildings and thus it is more useful to refine the trained model forthese systems to detect vehicles in low light or reflected lightconditions respectively. Further training the model of that system basedon images captured during its ongoing use allows the model to accountbetter for its surroundings, for example the landscape, nearbybuildings, flora etc. This may result in a more accurate model and moreconsistent detection of vehicles as the system becomes tailored to aparticular location of implementation.

Moreover, the conditions within any particular location ofimplementation will typically vary over the course of the day or over ayear. For example, a system may be required to detect and/or classifyvehicles in both daylight and night time conditions. The system may alsobe required to detect and/or classify vehicles in winter months whereice and snow are common, and in the summer months with bright sunshine.The further, localised and ongoing training provided in accordance withthe invention can beneficially accommodate this.

Detecting a vehicle involves determining whether a vehicle is present,e.g. within a certain range or in a certain zone. The range maycorrespond to the range in which the receiver can receive transmissionsfrom a vehicle and/or the field of view of the camera. Classifying avehicle involves determining at least one characteristic feature of avehicle, where the characteristic feature distinguishes the vehicle fromat least a plurality of other vehicles.

The data transmission may include a variety of different types of data.In a set of embodiments, the data transmission comprises a uniqueidentification code. The unique identification code may, for example, bethe license plate number of the vehicle.

The data transmission received from the vehicle may comprise otherinformation about the vehicle. This information may be transmitted inaddition to or instead of a unique identification code. In a set ofembodiments therefore, the data transmission comprises at least onevehicle parameter. For example, vehicle parameters may include about themodel, length, height or classification of the vehicle. The vehicleparameter may be used to classify the vehicle in accordance with one ormore of the features of the vehicle, e.g. type of vehicle (motorbike,car, lorry etc), size of vehicle (height, length, profile etc). Theclassification of a vehicle may be used in tolling applications tocorrectly charge the detected vehicle according to predetermined tollgroups, analysis of road use, dispatching of appropriate assistance etc.

Should the data transmission not comprise a vehicle parameter at all(e.g. containing no more detailed information about the vehicle than aunique identification code) or not contain all required parameters, itmay still be desirable to obtain any such ‘missing’ parameters fromelsewhere. In a set of embodiments, the system is arranged to send datafrom the data transmission to an external server and to receive from theserver at least one vehicle parameter (e.g. model, colour, length,height, classification). The external server, for example, may becontrolled by a vehicle licensing agency. In this way the processingsubsystem receives information resulting from data in the datatransmission which it can use for classification.

In a set of embodiments, in the second mode of operation identifying thevehicle comprises determining at least one vehicle parameter using thecaptured image.

In a set of the embodiments above the vehicle parameter is used by theprocessing subsystem in the first mode to train the model further bycorrelating the parameter to the captured image. This could be donewhether the parameter is received in the data transmission or from anexternal server.

As mentioned above, the data transmission may comprise a uniqueidentification code. This may comprise, or may be capable of beingmatched to, a license plate number for the vehicle. Systems inaccordance with the invention may additionally or alternatively bearranged to determine a license plate number. This could be done usingthe trained model but in a set of embodiments a dedicated imagerecognition algorithm is used. Such Automatic License Plate Recognitionalgorithms are well known per se in the art. In a set of embodiments thededicated image recognition algorithm uses the same captured image as isused by the trained model for detection/classification set out herein.The system could determine license plate numbers whenever a vehicle isdetected (e.g. for verification purposes) or only when no datatransmission is received.

It may be beneficial in certain applications to implement an additionalmode in which the data transmission is simply used for detecting and/orclassifying the vehicle (i.e. rather than further training the model).This may reduce the power consumption of the system, or allow the systemto continue to operate in a situation in which an image in not capturedor is too poor to be beneficial in further training (e.g. the camera isbroken, the image captured is partly obscured etc.). Similarly, whilstfurther training the model could be carried out substantially at thesame time the data is captured, this is not essential. For example thedata transmissions and corresponding images could be batch processed bythe system during quieter periods such as at night. Further training themodel could be carried out locally to where the camera and receiver arelocated or the data transmissions and corresponding images could betransmitted to a remote server.

The camera may be any suitable and desirable type of camera. Whilst thecamera may be arranged to capture any desirable form of electromagneticradiation (e.g. infra-red radiation), in a set of embodiments the camerais an optical camera. The camera may comprise any suitable or desirablelens, for example an ultra wide-angle (‘fish-eye’) lens. Implementing anultra wide-angle lens helps to achieve a wide panoramic or hemisphericalimage, which may improve the range for vehicle detection and/oridentification.

The camera may be installed in any suitable or desirable position, forexample, by the side of a road or on a gantry above a road. The camerais ideally positioned such that an unobscured and clear image of avehicle may be obtained.

In a set of embodiments, the system comprises a plurality of cameras.The plurality of cameras may be arranged at different positions and/orangles, such that multiple images may be captured of a vehicle fromvarious angles and positions (e.g. at different positions along a road).The plurality of cameras may comprise various types of cameras (e.g.optical cameras, infra-red cameras). The plurality of cameras maycapture images at different exposure levels (also called high dynamicrange imaging) to account for varying light conditions within the fieldof view. This may allow additional information to be obtained from theimages for detection and/or classification compared with implementingonly a single camera. A single common trained model may be used toanalyse images from the plurality of cameras or different cameras may beassociated with different trained models or different neural networks orsub-networks within a trained model.

In a set of embodiments, the camera(s) is arranged to capture aplurality of images. By capturing a plurality of images, an increasednumber of images are obtained for use in the further training of themodel, which may further improve the model. Similarly extra images mayimprove the chance of the trained model(s) successfully detecting and/orclassifying the vehicle in the second mode.

Alongside detecting and/or classifying a vehicle, it will typically beimportant to track the vehicle—e.g. to ensure that vehicles arecorrectly detected even if they are close together or take evasiveaction such as attempting to avoid detection by a tolling system. Thevehicle may be tracked in any suitable or desirable manner, howeverpreferably the vehicle is tracked using a plurality of (consecutive)images. A plurality of images may be captured of the vehicle usingeither a single camera over a period of time, or by multiple cameras atvarious locations along the route of the vehicle (e.g. forward andrearward facing). The first mode of operation may additionally includethe camera(s) capturing a plurality of images, and the processingsubsystem using the transmission and the images to train or furthertrain the model to track vehicles. The second mode of operation mayadditionally include the camera(s) capturing a plurality of images, andthe processing subsystem using the images and the trained model to trackthe vehicle in the images.

In some examples, tracking vehicles comprises determining whether one ormore vehicles is approaching, if so how many, and whether a givenvehicle is moving or stationary. Detecting that a vehicle is stationarymay indicate that there has been an accident or a break-down, and thismay trigger assistance to be sent to the vehicle. The vehicle trackingmay also allow the vehicle speed to be determined. Determining vehiclespeed may be used in applications in which speed is monitored to promoteor enforce speed limits.

The receiver may be arranged such that it only receives a signal from avehicle when the vehicle is within a predetermined distance of thereceiver (limited by the range of the communication signals).Alternatively a zone may be defined—e.g. by means of road markings.Communications may be one-way from the vehicle to the system such thatthe vehicle has just a transmitter and the system has just a receiver.Equally however communications may be two-way.

The receiver may be arranged to receive any suitable and desirable formof data transmission from a vehicle. A vehicle may comprise atransmitter or transceiver with an ‘on board unit’ (OBU). Thetransmission may comprise a Cooperative Intelligent Transport Systems(C-ITS) message, using ETSI-G5 or Cellular V2X technologies, whichincludes information which may uniquely identify the vehicle and/orprovide vehicle size or class information. Alternatively suchcommunications are dedicated short-range communications e.g. followingthe European Committee for Standardisation (CEN) dedicated short rangecommunication (DSRC) standard. In a set of embodiments therefore, thereceiver is a CEN DSRC receiver arranged to receive a CEN DSRCtransmission from a vehicle. The receiver may comprise part of a ‘roadside unit’ (RSU) which is arranged to communicate with the OBU in avehicle.

The arrangement of the system may allow a polling operation to beperformed and thus in a set of embodiments, a transmission from avehicle is acquired using a polling operation. The polling operation maycomprise transmitting a polling signal from the system (e.g. the RSU)and receiving a response signal transmitted by the vehicle (e.g.transmitted from the OBU) at the system. Preferably the response signalcomprises the data transmission. This process may help to conserve thepower source (e.g. battery) of a transceiver in the vehicle, as thetransceiver need not transmit a signal unless it receives the pollingsignal from the system (e.g. the transceiver is not required to transmita signal continually).

In a set of embodiments, the system comprises a plurality of receivers.The plurality of receivers may be arranged in any suitable or desirablemanner. For example, the plurality of receivers may be arranged atvarious positions along a road. In embodiments in which the systemcomprises a plurality of cameras, each of the plurality of receivers maybe located with a (corresponding) camera. This may ensure bettercorrelation between the transmitted signal and an image captured by acamera, for example helping to ensure that the vehicle captured in animage corresponds to the vehicle transmitting the signal.

The plurality of receivers may also comprise various types of receivers.The different types of receivers, may be arranged to receive variousdifferent types of transmission from a vehicle (e.g. CEN DSRC,Cooperative Intelligent Transport Systems (C-ITS), radio). Implementingvarious types of receivers may increase the number of transmissionsreceived by the system, and therefore the number of transmissions thatcan be used to refine the model in the first mode of operation of thesystem.

The processing subsystem may be located in the vicinity of the camera(s)and/or receiver(s), however the processing subsystem may also or insteadbe located remotely. In either of these embodiments, the receiver(s) andcamera(s) may be arranged to transmit data (e.g. the data transmissionreceived from a vehicle and the image captured) to the processingsubsystem. Especially in embodiments in which the processing subsystemis remote, preferably the transmission from the receiver and camera tothe processing subsystem is a wireless transmission.

In a set of embodiments, the processing subsystem is arranged topre-process the image based on external information separate from theimage and the data transmission. Pre-processing the image may help inthe first mode of operation to ensure that only appropriate images, e.g.those likely to be of high quality, are used to further train the model.

Pre-processing the images may help to improve the accuracy of the systemin detecting and/or classifying and/or tracking vehicles. Bypre-processing images, the model can be trained to account for differentconditions which may change how the model functions to detect, classifyor track a vehicle. In embodiments in which the model comprises a neuralnetwork, the relevant external information may be used alongside theimage and data transmission to refine the neural network. The neuralnetwork may then account for the relevant conditions when the system isoperating in the second mode to improve the detection and/orclassification and/or tracking of a vehicle.

In embodiments in which the model comprises a plurality of neuralnetworks, a specific neural network may be selectively trained dependingon the external information relating to the conditions during capture ofthe image. This may result in a plurality of neural networks whereineach is specialised to detect and/or classify and/or track a vehicle inparticular conditions, improving the accuracy of the system and method.

The Applicant has also recognised that it may be important forsuccessful further training of the model that the images used are of asufficiently high quality. In a set of embodiments therefore thepre-processing could be used to determine whether to use a capturedimage for training at all.

By pre-processing images using the external information, in the secondmode of operation an appropriate neural network of a plurality of neuralnetworks can be selected depending on the external information. This maylead to an improved detection and/or classification and/or tracking of avehicle, as an appropriate neural network can be selected which isoptimally trained to analyse the image captured in the specificconditions detected.

The captured image may be pre-processed based on any one or more of anumber of different types of external information. In an exemplary setof embodiments the image is categorised based on conditions in thelocality of the system (in particular the locality of the camera). Forexample, the image may be pre-processed based on the light conditionsand/or the weather conditions. Exemplary categories of such informationinclude raining, snowing, sunshine, daytime/night time and time of year.

To determine the conditions in the locality of the system, the system(e.g. the processing subsystem) may communicate with an external datasource to obtain information on one or more conditions in the localityof the system. The external data source could, for example, be a weatherdata server or sunrise/sunset calendar. Additionally or alternatively,in a set of embodiments, the system comprises a sensor subsystemarranged to monitor the condition(s). In a particular set ofembodiments, the sensor subsystem comprises a light sensor such as asolar panel or a photodiode arranged to provide an indication of a lightlevel. Similarly, the sensor subsystem could include a rain sensorand/or a temperature sensor arranged to provide an indication of theweather conditions.

BRIEF DESCRIPTION OF DRAWINGS

Some preferred embodiments of the present invention will now bedescribed, by way of example only, and with reference to theaccompanying drawings, in which:

FIG. 1 is an illustration of a system in accordance with an embodimentof the present invention;

FIG. 2 is an illustration of a system in accordance with anotherembodiment of the present invention;

FIGS. 3 a and 3 b are illustrations of a system in accordance withanother embodiment of the present invention;

FIG. 4 is a flowchart illustrating a method of training and implementinga model in accordance with an embodiment of the present invention; and

FIGS. 5 and 6 are exemplary captured images which have been classifiedby a system in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

FIG. 1 illustrates a system in accordance with an embodiment of thepresent invention. The system 200 shown in FIG. 1 comprises a singlegantry 208. The gantry 208 includes the following mounted components: aCEN DSRC transceiver 202, a camera 204, a processing subsystem 206, acellular data transceiver 207 and a condition monitoring system 212. Thearrow shown indicates the direction in which the vehicle is driving.

The CEN DSRC transceiver 202 is arranged to communicate with a CEN DSRCtag (not shown) in a vehicle 210. As is known per se in the art, the CENDSRC tag transmits data identifying the vehicle to the CEN DSRCtransceiver 202. This information includes a unique identificationnumber (e.g. the license plate number) as well as information such asthe size and class of the vehicle 210. However, it is appreciated thatnot all vehicles may possess a working CEN DSRC tag, therefore a systemwhich relies solely on receiving such identifying information would beineffective at detecting, identifying and/or classifying all vehicles.Therefore, as will be described below, the system 200 also includesimage recognition capability to detect and/or identify a vehicle 210from which no transmission is received.

The CEN DSRC tag may not transmit a data transmission until a pollingtransmission is received by the tag. To account for such circumstances,the CEN DSRC transceiver 202 periodically transmits a polling signal.Any suitable equipped vehicles 210 should then transmit a datatransmission to the system 200 in response.

The camera 204 is arranged to capture images along the road on which thesystem 200 is installed. The captured images are processed by theprocessing subsystem 206. The processing subsystem 206 implements, andin some circumstances further trains, a trained model comprising aconvolutional neural network as discussed in more detail below inrelation to FIG. 3 .

The system 200 shown in FIG. 1 further includes an environmentalmonitoring system 212. The monitoring system 212 contains a light sensorand rainfall sensor for determining the environmental conditions in thelocality of the system 200. The information from the monitoring system212 is used to help categorise the images captured by the camera 204,which is also discussed in more detail in relation to FIG. 3 . Theenvironmental conditions could alternatively be requested from otherexternal sources (e.g. a weather service) as shown in FIG. 2 . In eithercase the processing subsystem 206 may also be programmed withinformation about the camera geometry such that it can assess whatenvironmental conditions may impair the image capture process. (e.g. awest-facing camera may be affected in evening at a time corresponding tosunset).

The system 200 communicates with an external server 214 via the cellulardata transceiver 207 which connects to a public mobile data network.Alternatively a wired connection to a fixed network, e.g. comprisingfiber optic cable, could be provided. The external server 214 shown inFIG. 1 is for example a cloud-based database compiled by a licensingagency which allows a unique identifying number (e.g. a license platenumber) of a vehicle to be used to look up additional information aboutthe vehicle such as the model, colour, length, height, classification ofthe vehicle.

FIG. 2 illustrates a system in accordance with another embodiment of thepresent invention. Similarly to the system 200 shown in FIG. 1 , thesystem 250 shown in FIG. 2 includes a camera 254 arranged to captureimages along the road on which it is installed and a CEN DSRCtransceiver 252 arranged to communicate with a CEN DSRC tag (not shown)in a vehicle 260 to receive data transmissions as previously described.The camera 254 and the CEN DSRC receiver 252 are mounted on a gantry258.

However, in contrast to the embodiment shown in FIG. 1 , the processingsubsystem 256 which implements and further trains the neural network islocated remotely. In FIG. 2 , the processing subsystem 256 is part of aserver system 270. The server system 270 additionally comprises aweather data server 280 and a vehicle licensing agency server 290. Theweather data server 280 provide information on the weather conditions inthe locality of the system. The vehicle licensing agency server 290provides additional information about the vehicle 260 associated with aunique identification number such as the model, colour, length, height,classification of the vehicle 260.

Operation of the embodiments of FIGS. 1 and 2 will now be described withreference to FIG. 4 which is a flowchart of a method of detecting andidentifying vehicles using the system shown in FIGS. 1 and 2 . In step402, a camera 204, 254 captures an image. The camera 204, 254 may bearranged to continuously capture images.

In the system 200 shown in FIG. 1 , a wired or wireless connection isused to communicate the captured images from the camera 204 to theprocessing subsystem 206. For system 250 shown in FIG. 2 , whichincludes a remote processing subsystem 256, the method comprises anadditional step of transmitting the captured images to the remoteprocessing subsystem 256 (or more generally the server system 270). Thetransmitter 265 transmits the image via the mobile communication networkto the remote processing subsystem 256.

In step 404, the image captured by the camera 204, 254 is pre-processedby the processing subsystem 206, 256 based on information obtained fromthe environmental condition monitoring system 212 or the weather dataserver 280. As will be explained below, pre-processing using suchexternal information is used to select the most appropriate neuralnetwork or to enhance the further training which can be provided.

In a simple form, this pre-processing could be dependent on lightconditions under which the image was captured where the conditionmonitoring system 212 provides information on the light levels detected.The processing subsystem 206 uses the information provided by thecondition monitoring system to label the images by the light levelsunder which they were taken.

In a more complex form, the categories could be expanded further.Further information regarding the weather conditions, obtained from themonitoring system 212 or via communication with an external server 214,270 could be used to provide more detailed categories. In the system 250of FIG. 2 for example, a local weather report obtained from the weatherdata server 280 may be used by the processing subsystem 256 to label theimage captured by the camera 254 according to categories such assunshine, cloud, rain and snow.

In step 406, the system 200, 250 determines whether the CEN DSRCtransceiver 202, 252 has received a data transmission signal from a DSRCtag in the vehicle 210, 260.

The system 200, 250 may determine in step 406 that a transmission is notsuccessfully received. This may be either due to either no vehicle beingpresent e.g. in the range of the CEN DSRC receiver 202, 252 and/or inthe camera image, or a vehicle 210, 260 passing through the system whichhas no CEN DSRC tag, or the transmission suffering too muchinterference.

In step 416, the processing subsystem 206, 256 uses the model and thecaptured image to determine whether a vehicle is present. In exampleswhere the model comprises a plurality of convolutional neural networks,the label may be used to choose an appropriate CNN. More specificallyseveral steps of pre-processing are applied to reduce the data volumeand the number of parameters. The CNN has multiple convolution layerseffectively creating multiple derived images from the input image. Theconvolution layer (or process) employs a sliding window with weightsthat is applied to a subsection of the image to produce a weightedaverage value for each subsection. The convolution process results in anoutput image where the resulting size is independent of the input imagesize (i.e. camera resolution). Before the convolution process, standardmachine learning procedures are employed including converting colourinput images to grayscale, cropping areas where no vehicle should bedetected, and normalizing grayscale values applied. Convolution filtersare arranged to detect edges and corners in the image. This means thatthe inner layers of the model will work on edges and corners in theoriginal image and not areas of uniform colour.

To reduce the size of the input data and thus the model parameters,another non-linear convolution step is applied to extract a strongestimage feature, e.g. an edge or corner, in each of the subsections. Thelarge amount of images with known classification which are provided byembodiments of the invention make it possible to benefit from additionallayers compared to what might typically be used.

In step 418 the processing subsystem 206, 256 determines whether avehicle has been detected.

If at step 418 it is determined that there is no vehicle present in thecaptured image, the method starts again from step 402.

Alternatively, at step 418, if the system 200, 250 determines a vehicle210, 260 has been detected, i.e. a vehicle 210, 260 is present in theimage, the processing subsystem 206, 256 uses the captured image, thepre-processing label and the model to classify the detected vehicle 210,260 (step 420). In examples where the model comprises a plurality ofneural networks, the label may be used in this step to select anappropriate neural network. Unique identification may also be carriedout using a license plate recognition algorithm.

However, in some applications or instances, it may not be possible ornecessary to correctly uniquely identify the vehicle 210, 260 from thecaptured image. For instance, whilst the system may include a licenseplate recognition algorithm, it may be unable to determine the licenseplate e.g. owing to the resolution of the image, the size of the numberplate in the image. In other applications, the system 200, 250 may notbe designed to identify a license plate. In such instances, the system200, 250 may simply classify the detected vehicle 210, 260 based onrelevant parameters (e.g. the length and/or width of the vehicle).Classifying the detected vehicle 210, 260 based on the relevantparameters may be sufficient, for example, to classify the vehicle inthe correct toll tariff so that the vehicle 210, 260 can be chargedappropriately at a tolling station. The classification of the detectedvehicle may also be sufficient to provide data for analysis of road use,for example how much each classification of vehicle (e.g. motorcycle,car, lorry) uses the road.

Alternatively, if a transmission has been successfully received at step406, step 408 is implemented. In step 408 the system 200, 250 identifiesthe vehicle 210, 260 based on the transmission received. The signalreceived by the CEN DSRC transceiver 202 includes a uniqueidentification code, therefore providing a unique identification of thevehicle. This unique identification code could be the license platenumber or some other unique identifier which can be used to determinethe license number from the licensing agency server 290.

The system 200, 250 then proceeds in step 410 to determine relevantinformation and parameters relating to the vehicle 210, 260 using thetransmission received in step 408. Whilst the transmission received instep 408 may already include additional information such as the model,classification (for example whether car, lorry, motorcycle), dimensionsof the vehicle 210, 260, it may instead be necessary for such relevantinformation to be obtained by communicating with a remote server 214,290. The classification and/or dimensions of the detected vehicle 210,260 determined in step 410 may be used in tolling applications to levy acharge suitably according to predetermined toll tariffs. This may allowfor an owner of the vehicle 210, 260 to automatically be charged thecorrect toll at the toll gate, without relying on the driver or anoperator to select the correct toll tariff for the vehicle.Additionally, an account associated with the unique identificationnumber of the vehicle 210, 260 is automatically debited with a suitablecharge based on the classification of the vehicle 210, 260. Accountinformation may be held in a payment server (not shown). In the system250 of FIG. 2 , a payment server may be included in the server system270.

The additional information and parameters relating to the vehicle 210,260 determined in step 410 may also be used to analyse road use. Thiscould be performed by a component of the system 200, 250 such as theserver system 270 and/or analysis could take place in a separate remotesystem.

In step 412, the captured image is labelled by the processing subsystem206, 256 with the relevant information, such as the model,classification and dimensions of the vehicle 210, 260 determined in step410. The image may additionally be labelled with the category determinedin step 404.

In step 414, the labelled image is used to train the model further. Fora model that comprises a neural network, the further training involvesstrengthening certain pathways within the neural network to create amore accurate model. In particular, using a labelled image may help tostrengthen links and pathways in the neural network. While training themodel, a relevant cost function is used to measure the n-dimensionaldistance between the known classification vector and the output from theneural network. In general, neural networks will have better performanceas additional layers are added, but this requires a larger trainingdataset. This is exploited in accordance with the invention where acontinuously growing training set is available.

In examples where the model comprises a plurality of neural networks,the label assigned to the image in step 404 may be used to select asuitable neural network specific to the label (e.g. environmentalconditions) to further refine. As different neural networks are providedwith different images according to their categorisation, differentpathways will be strengthened in different neural networks. Over thecourse of multiple iterations of further refinement, the differentneural networks may become increasing different as different pathwaysare strengthened.

Whilst step 410, step 412 and step 414 may take place sequentiallydirectly after step 408, these steps may occur at a later point. Theidentification of the vehicle based on the transmission may be stored,e.g. in a memory within the system or in an external cloud storage, andsteps 410 to 414 completed at a later point. Alternatively, only step412 or step 414 may be delayed. For example, step 414 may only beimplemented once a significant number of labelled images have beenacquired by the system 200, 250. The training of the model may thenoccur in batches, for example once a day, rather than continuouslythroughout the day.

A delay in implementing one or more of steps 410 to 414 may occur when,for example, the road on which the system 200, 250 is implemented isexperiencing a high volume of traffic and the time delay betweenconsecutive vehicles 210, 260 is small, helping to reduce the overallworkload of the system during this period.

Typically the method is repeated continually, many times a second

Repeating the steps set out in FIG. 4 enables the tracking of a detectedvehicle 210, 260, by identifying the same vehicle to be present inconsecutive images captured. This may allow a system 200, 250additionally to determine the trajectory of a detected vehicle and itsspeed e.g. to ensure accurate counting of vehicles and capturingvehicles that have only one license plate or different license plates onthe front and rear. This may also allow, for example, determination thata vehicle has stopped, and to request assistance for the stoppedvehicle.

In an alternative embodiment where polling is not continually performed,following the detection of a vehicle in step 418, the system performs apolling operation—i.e. the DSRC transceiver 202, 252 transmits a pollingsignal which is received by any CEN DSRC tags in a vehicle 210, 260. ACEN DSRC tag then responds with an identifying transmission. In thisembodiment, the model is used to detect the vehicle 210, 260, theidentifying transmission is used to identify the detected vehicle, andstep 420 is unnecessary. Should no response be received by the CEN DSRCtransceiver then step 420 is conducted as described above.

FIGS. 5 and 6 show two examples of a detected vehicle being classifiedfrom a captured image. In the cases of the detected vehicles shown inFIGS. 5 and 6 , it may not be possible to identify the vehicle (e.g.determine the license plate number of the vehicle) from the capturedimage. In FIGS. 5 and 6 , the license plate number is obscured by thelights of the vehicle and by the resolution of the captured image. Themodel determines the rough outline of the vehicle (as shown by thebordering box around the vehicle) and then classifies the vehicle. InFIG. 5 the vehicle has been classified as a small car, and in FIG. 6 thevehicle has been classified as a truck. Such classification are usefulin tolling applications, or for gaining general information about theuse of the area (e.g. road) which the system monitors. A numericalconfidence level is also assigned to the classification. The confidencelevel ranges from 0 to 1. A confidence level of 0 indicates noconfidence in the classification of the vehicle, and a confidence levelof 1 indicates complete confidence in the classification of the vehicle.In practise, the confidence level will assume values between 0 and 1,and not the value of one of these extremities.

Whilst the systems 200 and system 250 of the first and secondembodiments only comprise a single camera 204, a similar system 300 isshown in FIGS. 3 a and 3 b which comprises two cameras 304, 305. Thefirst camera 304 and the second camera 305 are positioned on the gantry308 such that they can obtain images in different directions. In theparticular arrangement shown in FIGS. 3 a and 3 b , the first camera 304may capture an image of the front of the vehicle 310 as the vehicle 310progresses toward the gantry 308 (as shown in FIG. 1 ) and the secondcamera 305 can capture an image of the rear of the vehicle 310 as thevehicle 310 moves away from the gantry 208. This may improve thetracking capabilities of the system 300, or allow more information on avehicle 310 to be obtained as is explained below.

Similarly to the embodiment shown in FIG. 1 , in FIGS. 3 a and 3 b theimages captured by both cameras 304, 305 are communicated to aprocessing system 306. The processing subsystem 306 refines the modeldifferently depending on whether an image received is captured by thefirst camera 304 or the second camera 305. This allows differences inthe surroundings, for example the landscape, nearby buildings, floraetc., in the field of view of the first camera 304 and second camera 305to be accounted for in the model. This also enables differences in theportion (e.g. front and rear) of a vehicle 310 captured in an image tobe accounted for in the model and to capture images including the frontand rear license plates.

In the system 300, which include two cameras 304, 305, both cameras 304,305 typically continuously captures images. Tracking of the vehicle 310can be performed over an extended section of road on which the system isinstalled, as the vehicle 310 can be tracked first using consecutiveimages captured by the first camera 304 and then tracked usingconsecutive images captured by the second camera 305.

Furthermore in this embodiment, the processing subsystem 306additionally labels an image in step 412 as being captured by both thefirst camera 304 and the second camera 305. Therefore, in step 414, themodel is further refined using this information. In examples where themodel comprises a plurality of neural networks, labelling an imageaccording to the capturing camera 304, 305 may allow a suitable neuralnetwork for a specific camera 304, 305 to be selected to refine. It willbe appreciated that having distinct neural networks associated with eachcamera may be beneficial in allowing them to become adapted to theparticular characteristics of their location and outlook and forrecognising the front or rear of a vehicle respectively.

Thus it will be appreciated by those skilled in the art that thespecific embodiments of the inventive concepts described herein providesystems and methods suitable for training a model to detect and/orclassify vehicles in situ. This may provide significant benefits overknown devices. It will further be appreciated that many variations ofthe specific arrangements described here are possible within the scopeof the invention.

1. A vehicle detection system comprising: a receiver arranged to receivedata transmissions from a vehicle; a camera arranged to capture an imageof said vehicle; and a processing subsystem arranged to implement atrained model for detecting and/or classifying vehicles in an image;wherein the vehicle detection system is arranged to use informationresulting from said data transmissions from the vehicle together withsaid captured images of the vehicle to provide data for improving thetrained model, the vehicle detection system being further arranged touse the trained model to detect and/or classify the vehicle in the imageif no data transmission is received.
 2. The vehicle detection system ofclaim 1 wherein the system is arranged to operate in at least a firstmode wherein: the camera captures an image of a vehicle and the receiverreceives a data transmission from the vehicle; and the processingsubsystem is arranged to use information resulting from the datatransmission received and the image to train the model further; and asecond mode wherein: the camera captures an image of a vehicle and thereceiver does not receive a data transmission; and the processingsubsystem is arranged to use the image and the trained model to detectand/or classify the vehicle in the image.
 3. The vehicle detectionsystem of claim 1 wherein the trained model comprises a neural network.4. The vehicle detection system of claim 2 wherein the neural networkcomprises a convolutional neural network.
 5. The vehicle detectionsystem of claim 3 wherein the convolutional neural network is arrangedto provide an output image where the resulting size is independent ofthe input image size.
 6. The vehicle detection system of claim 5 whereinthe convolutional neural network is arranged to apply a non-linearconvolution step to extract a strongest image feature in each of aplurality of subsections.
 7. The vehicle detection system of claim 1wherein the data transmission comprises a unique identification code. 8.The vehicle detection system of claim 1 wherein the data transmissioncomprises at least one vehicle parameter.
 9. The vehicle detectionsystem of claim 1 arranged to send data from the data transmission to anexternal server and to receive from the server at least one vehicleparameter.
 10. The vehicle detection system of claim 1 arranged, in thesecond mode of operation, to identify the vehicle by determining atleast one vehicle parameter using the captured image.
 11. The vehicledetection system of claim 8 wherein the processing subsystem isarranged, in the first mode, to use the vehicle parameter to train themodel further by correlating the vehicle parameter to the capturedimage.
 12. The vehicle detection system of claim 1 comprising aplurality of cameras.
 13. The vehicle detection system of claim 1arranged to track the vehicle using a plurality of images.
 14. Thevehicle detection system of claim 1 arranged to acquire a transmissionfrom the vehicle using a polling operation.
 15. The vehicle detectionsystem of claim 1 wherein the processing subsystem is arranged topre-process the image based on external information separate from theimage and the data transmission.
 16. The vehicle detection system ofclaim 15 comprising a plurality of neural networks and arranged toselectively train a specific neural network depending on the externalinformation relating to conditions during capture of the image.
 17. Thevehicle detection system of claim 1 arranged to pre-process the capturedimage based on conditions in a locality of the system.
 18. The vehicledetection system of claim 1 comprising a sensor subsystem arranged tomonitor conditions in a locality of the system.
 19. A method ofdetecting vehicles, the method comprising: capturing an image of avehicle using a camera; using a receiver to listen for a datatransmission from the vehicle; determining whether said datatransmission has been received from the vehicle; if said datatransmission is received from the vehicle, using said image to train atrained model further to detect and/or classify vehicles, and if no datatransmission is received from the vehicle, using the image and thetrained model to detect and/or classify the vehicle in the image.
 20. Anon-transitory computer-readable medium having stored thereon programinstructions that, upon execution by a computing device, cause thecomputing device to: capture an image of a vehicle using a camera;listen for a data transmission from the vehicle; determine whether saiddata transmission has been received from the vehicle; if said datatransmission is received from the vehicle, use said image to train atrained model further to detect and/or classify vehicles, and if no datatransmission is received from the vehicle, use the image and the trainedmodel to detect and/or classify the vehicle in the image.